On Tackling Flash Crowds with URL Shorteners and Examining User Behavior after Great East Japan Earthquake

Several web sites providing disaster-related information failed repeatedly after the Great East Japan Earthquake, due to flash crowds caused by Twitter users. Twitter, which was intensively used for information sharing in the aftermath of the earthquake, relies on URL shorteners like bit.ly to offset its strict limit on message length. In order to mitigate the flash crowds, we examine the current Web usage and find that URL shorteners constitute a layer of indirection; a significant part of Web traffic is guided by them. This implies that flash crowds can be controlled by URL shorteners. We developed a new URL shortener, named rcdn.info, just after the earthquake; rcdn.info redirects users to a replica created on a CoralCDN, if the original site is likely to become overloaded. This surprisingly simple solution worked very well in the emergency. We also conduct a thorough analysis of the request log and present several views that capture user behavior in the emergency from various aspects. Interestingly, the traffic significantly grew up at previously unpopular (i.e., small) sites during the disaster; this traffic shift could lead to the failure of several sites. Finally, we show that rcdn.info has great potential in mitigating such failures. We believe that our experience will help the research community tackle future disasters. key words: flash crowd, URL shortener, CDN, disaster, Great East Japan Earthquake